{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
        "\n",
        "Licensed under the MIT License."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/ml-frameworks/tensorflow/hyperparameter-tune-and-warm-start-with-tensorflow/hyperparameter-tune-and-warm-start-with-tensorflow.png)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Warm start hyperparameter tuning\n",
        "In this tutorial, you will learn how to warm start a hyperparameter tuning run from a previous tuning run."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's get started. First let's import some Python libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "nbpresent": {
          "id": "c377ea0c-0cd9-4345-9be2-e20fb29c94c3"
        }
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "import numpy as np\n",
        "import os\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "nbpresent": {
          "id": "edaa7f2f-2439-4148-b57a-8c794c0945ec"
        }
      },
      "outputs": [],
      "source": [
        "import azureml\n",
        "from azureml.core import Workspace\n",
        "\n",
        "# check core SDK version number\n",
        "print(\"Azure ML SDK Version: \", azureml.core.VERSION)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Diagnostics\n",
        "Opt-in diagnostics for better experience, quality, and security of future releases."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "Diagnostics"
        ]
      },
      "outputs": [],
      "source": [
        "from azureml.telemetry import set_diagnostics_collection\n",
        "\n",
        "set_diagnostics_collection(send_diagnostics=True)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Initialize workspace\n",
        "Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "ws = Workspace.from_config()\n",
        "print('Workspace name: ' + ws.name, \n",
        "      'Azure region: ' + ws.location, \n",
        "      'Subscription id: ' + ws.subscription_id, \n",
        "      'Resource group: ' + ws.resource_group, sep = '\\n')"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "nbpresent": {
          "id": "59f52294-4a25-4c92-bab8-3b07f0f44d15"
        }
      },
      "source": [
        "## Create an Azure ML experiment\n",
        "Let's create an experiment named \"tf-mnist\" and a folder to hold the training scripts. The script runs will be recorded under the experiment in Azure."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "nbpresent": {
          "id": "bc70f780-c240-4779-96f3-bc5ef9a37d59"
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core import Experiment\n",
        "\n",
        "script_folder = './tf-mnist'\n",
        "os.makedirs(script_folder, exist_ok=True)\n",
        "\n",
        "exp = Experiment(workspace=ws, name='tf-mnist-2')"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "nbpresent": {
          "id": "defe921f-8097-44c3-8336-8af6700804a7"
        }
      },
      "source": [
        "## Download MNIST dataset\n",
        "In order to train on the MNIST dataset we will first need to download it from Yan LeCun's web site directly and save them in a `data` folder locally."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import urllib\n",
        "data_folder = 'data'\n",
        "os.makedirs(data_folder, exist_ok=True)\n",
        "\n",
        "urllib.request.urlretrieve('https://azureopendatastorage.blob.core.windows.net/mnist/train-images-idx3-ubyte.gz',\n",
        "                           filename=os.path.join(data_folder, 'train-images-idx3-ubyte.gz'))\n",
        "urllib.request.urlretrieve('https://azureopendatastorage.blob.core.windows.net/mnist/train-labels-idx1-ubyte.gz',\n",
        "                           filename=os.path.join(data_folder, 'train-labels-idx1-ubyte.gz'))\n",
        "urllib.request.urlretrieve('https://azureopendatastorage.blob.core.windows.net/mnist/t10k-images-idx3-ubyte.gz',\n",
        "                           filename=os.path.join(data_folder, 't10k-images-idx3-ubyte.gz'))\n",
        "urllib.request.urlretrieve('https://azureopendatastorage.blob.core.windows.net/mnist/t10k-labels-idx1-ubyte.gz',\n",
        "                           filename=os.path.join(data_folder, 't10k-labels-idx1-ubyte.gz'))"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "nbpresent": {
          "id": "c3f2f57c-7454-4d3e-b38d-b0946cf066ea"
        }
      },
      "source": [
        "## Show some sample images\n",
        "Let's load the downloaded compressed file into numpy arrays using some utility functions included in the `utils.py` library file from the current folder. Then we use `matplotlib` to plot 30 random images from the dataset along with their labels."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "nbpresent": {
          "id": "396d478b-34aa-4afa-9898-cdce8222a516"
        }
      },
      "outputs": [],
      "source": [
        "from utils import load_data\n",
        "\n",
        "# note we also shrink the intensity values (X) from 0-255 to 0-1. This helps the model converge faster.\n",
        "X_train = load_data(os.path.join(data_folder, 'train-images-idx3-ubyte.gz'), False) / 255.0\n",
        "X_test = load_data(os.path.join(data_folder, 't10k-images-idx3-ubyte.gz'), False) / 255.0\n",
        "y_train = load_data(os.path.join(data_folder, 'train-labels-idx1-ubyte.gz'), True).reshape(-1)\n",
        "y_test = load_data(os.path.join(data_folder, 't10k-labels-idx1-ubyte.gz'), True).reshape(-1)\n",
        "\n",
        "# now let's show some randomly chosen images from the training set.\n",
        "count = 0\n",
        "sample_size = 30\n",
        "plt.figure(figsize = (16, 6))\n",
        "for i in np.random.permutation(X_train.shape[0])[:sample_size]:\n",
        "    count = count + 1\n",
        "    plt.subplot(1, sample_size, count)\n",
        "    plt.axhline('')\n",
        "    plt.axvline('')\n",
        "    plt.text(x=10, y=-10, s=y_train[i], fontsize=18)\n",
        "    plt.imshow(X_train[i].reshape(28, 28), cmap=plt.cm.Greys)\n",
        "plt.show()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Create a FileDataset\n",
        "A FileDataset references single or multiple files in your datastores or public urls. The files can be of any format. FileDataset provides you with the ability to download or mount the files to your compute. By creating a dataset, you create a reference to the data source location. If you applied any subsetting transformations to the dataset, they will be stored in the dataset as well. The data remains in its existing location, so no extra storage cost is incurred. [Learn More](https://aka.ms/azureml/howto/createdatasets)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.dataset import Dataset\n",
        "web_paths = ['https://azureopendatastorage.blob.core.windows.net/mnist/train-images-idx3-ubyte.gz',\n",
        "             'https://azureopendatastorage.blob.core.windows.net/mnist/train-labels-idx1-ubyte.gz',\n",
        "             'https://azureopendatastorage.blob.core.windows.net/mnist/t10k-images-idx3-ubyte.gz',\n",
        "             'https://azureopendatastorage.blob.core.windows.net/mnist/t10k-labels-idx1-ubyte.gz'\n",
        "            ]\n",
        "dataset = Dataset.File.from_files(path = web_paths)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Use the register() method to register datasets to your workspace so they can be shared with others, reused across various experiments, and referred to by name in your training script.\n",
        "You can try get the dataset first to see if it's already registered."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "dataset_registered = False\n",
        "try:\n",
        "    temp = Dataset.get_by_name(workspace = ws, name = 'mnist-dataset')\n",
        "    dataset_registered = True\n",
        "except:\n",
        "    print(\"The dataset mnist-dataset is not registered in workspace yet.\")\n",
        "\n",
        "if not dataset_registered:\n",
        "    dataset = dataset.register(workspace = ws,\n",
        "                               name = 'mnist-dataset',\n",
        "                               description='training and test dataset',\n",
        "                               create_new_version=True)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Create or Attach existing AmlCompute\n",
        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
        "\n",
        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If we could not find the cluster with the given name, then we will create a new cluster here. We will create an `AmlCompute` cluster of `Standard_NC6s_v3` GPU VMs. This process is broken down into 3 steps:\n",
        "1. create the configuration (this step is local and only takes a second)\n",
        "2. create the cluster (this step will take about **20 seconds**)\n",
        "3. provision the VMs to bring the cluster to the initial size (of 1 in this case). This step will take about **3-5 minutes** and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
        "from azureml.core.compute_target import ComputeTargetException\n",
        "\n",
        "# choose a name for your cluster\n",
        "cluster_name = \"hd-cluster\"\n",
        "\n",
        "try:\n",
        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
        "    print('Found existing compute target')\n",
        "except ComputeTargetException:\n",
        "    print('Creating a new compute target...')\n",
        "    compute_config = AmlCompute.provisioning_configuration(vm_size='Standard_NC6s_v3',\n",
        "                                                           max_nodes=4)\n",
        "\n",
        "    # create the cluster\n",
        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
        "\n",
        "# can poll for a minimum number of nodes and for a specific timeout. \n",
        "# if no min node count is provided it uses the scale settings for the cluster\n",
        "compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
        "\n",
        "# use get_status() to get a detailed status for the current cluster. \n",
        "print(compute_target.get_status().serialize())"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now that you have created the compute target, let's see what the workspace's `compute_targets` property returns. You should now see one entry named 'gpu-cluster' of type `AmlCompute`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "compute_targets = ws.compute_targets\n",
        "for name, ct in compute_targets.items():\n",
        "    print(name, ct.type, ct.provisioning_state)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Copy the training files into the script folder\n",
        "The TensorFlow training script is already created for you. You can simply copy it into the script folder, together with the utility library used to load compressed data file into numpy array."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import shutil\n",
        "\n",
        "# the training logic is in the tf_mnist.py file.\n",
        "shutil.copy('./tf_mnist.py', script_folder)\n",
        "\n",
        "# the utils.py just helps loading data from the downloaded MNIST dataset into numpy arrays.\n",
        "shutil.copy('./utils.py', script_folder)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "nbpresent": {
          "id": "2039d2d5-aca6-4f25-a12f-df9ae6529cae"
        }
      },
      "source": [
        "## Construct neural network in TensorFlow\n",
        "In the training script `tf_mnist.py`, it creates a very simple DNN (deep neural network), with just 2 hidden layers. The input layer has 28 * 28 = 784 neurons, each representing a pixel in an image. The first hidden layer has 300 neurons, and the second hidden layer has 100 neurons. The output layer has 10 neurons, each representing a targeted label from 0 to 9.\n",
        "\n",
        "![DNN](nn.png)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Azure ML concepts  \n",
        "Please note the following three things in the code below:\n",
        "1. The script accepts arguments using the argparse package. In this case there is one argument `--data_folder` which specifies the file system folder in which the script can find the MNIST data\n",
        "```\n",
        "    parser = argparse.ArgumentParser()\n",
        "    parser.add_argument('--data_folder')\n",
        "```\n",
        "2. The script is accessing the Azure ML `Run` object by executing `run = Run.get_context()`. Further down the script is using the `run` to report the training accuracy and the validation accuracy as training progresses.\n",
        "```\n",
        "    run.log('training_acc', np.float(acc_train))\n",
        "    run.log('validation_acc', np.float(acc_val))\n",
        "```\n",
        "3. When running the script on Azure ML, you can write files out to a folder `./outputs` that is relative to the root directory. This folder is specially tracked by Azure ML in the sense that any files written to that folder during script execution on the remote target will be picked up by Run History; these files (known as artifacts) will be available as part of the run history record."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The next cell will print out the training code for you to inspect it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "with open(os.path.join(script_folder, './tf_mnist.py'), 'r') as f:\n",
        "    print(f.read())"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Create an environment\n",
        "\n",
        "In this tutorial, we will use one of Azure ML's curated TensorFlow environments for training. [Curated environments](https://docs.microsoft.com/azure/machine-learning/how-to-use-environments#use-a-curated-environment) are available in your workspace by default. Specifically, we will use the TensorFlow 2.0 GPU curated environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core import Environment\n",
        "\n",
        "tf_env = Environment.get(ws, name='azureml-tensorflow-2.11-cuda11')"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Configure the training job\u00c2\u00b6\n",
        "Create a ScriptRunConfig object to specify the configuration details of your training job, including your training script, environment to use, and the compute target to run on."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core import ScriptRunConfig\n",
        "\n",
        "args = ['--data-folder', dataset.as_mount(),\n",
        "        '--batch-size', 64,\n",
        "        '--first-layer-neurons', 256,\n",
        "        '--second-layer-neurons', 128,\n",
        "        '--learning-rate', 0.01]\n",
        "\n",
        "src = ScriptRunConfig(source_directory=script_folder,\n",
        "                      script='tf_mnist.py',\n",
        "                      arguments=args,\n",
        "                      compute_target=compute_target,\n",
        "                      environment=tf_env)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Submit job to run\n",
        "Submit the ScriptRunConfig to an Azure ML experiment to kick off the execution."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "run = exp.submit(src)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.widgets import RunDetails\n",
        "RunDetails(run).show()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Intelligent hyperparameter tuning\n",
        "Now that we have trained the model with one set of hyperparameters, we can tune the model hyperparameters to optimize model performance. First let's define the parameter space using random sampling. Typically, the hyperparameter exploration process is painstakingly manual, given that the search space is vast and evaluation of each configuration can be expensive.\n",
        "\n",
        "Azure Machine Learning allows you to automate hyperparameter exploration in an efficient manner, saving you significant time and resources. You specify the range of hyperparameter values and a maximum number of training runs. The system then automatically launches multiple simultaneous runs with different parameter configurations and finds the configuration that results in the best performance, measured by the metric you choose. Poorly performing training runs are automatically early terminated, reducing wastage of compute resources. These resources are instead used to explore other hyperparameter configurations."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We start by defining the hyperparameter space. In this case, we will tune 4 hyperparameters - '--batch-size', '--first-layer-neurons', '--second-layer-neurons' and '--learning-rate'. For each of these hyperparameters, we specify the range of values they can take. In this example, we will use Random Sampling to randomly select hyperparameter values from the defined search space."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.train.hyperdrive import RandomParameterSampling, choice, loguniform\n",
        "\n",
        "ps = RandomParameterSampling(\n",
        "    {\n",
        "        '--batch-size': choice(32, 64, 128),\n",
        "        '--first-layer-neurons': choice(16, 64, 128, 256, 512),\n",
        "        '--second-layer-neurons': choice(16, 64, 256, 512),\n",
        "        '--learning-rate': loguniform(-6, -1)\n",
        "    }\n",
        ")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Next, we will create a new ScriptRunConfig without the above parameters since they will be passed in later. Note we still need to keep the `data-folder` parameter since that's not a hyperparamter we will sweep."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "args = ['--data-folder', dataset.as_mount()]\n",
        "\n",
        "src = ScriptRunConfig(source_directory=script_folder,\n",
        "                      script='tf_mnist.py',\n",
        "                      arguments=args,\n",
        "                      compute_target=compute_target,\n",
        "                      environment=tf_env)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Next we will define an early termnination policy. This will terminate poorly performing runs automatically, reducing wastage of resources and instead efficiently using these resources for exploring other parameter configurations. In this example, we will use the `TruncationSelectionPolicy`, truncating the bottom performing 25% runs. It states to check the job every 2 iterations. If the primary metric (defined later) falls in the bottom 25% range, Azure ML terminate the job. This saves us from continuing to explore hyperparameters that don't show promise of helping reach our target metric."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.train.hyperdrive import TruncationSelectionPolicy\n",
        "policy = TruncationSelectionPolicy(evaluation_interval=2, truncation_percentage=25)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we are ready to configure a run configuration object, and specify the primary metric `validation_acc` that's recorded in your training runs. If you go back to visit the training script, you will notice that this value is being logged after every epoch (a full batch set). We also want to tell the service that we are looking to maximizing this value. We also set the number of samples to 15, and maximal concurrent job to 4, which is the same as the number of nodes in our computer cluster."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.train.hyperdrive import HyperDriveConfig, PrimaryMetricGoal\n",
        "htc = HyperDriveConfig(run_config=src, \n",
        "                       hyperparameter_sampling=ps, \n",
        "                       policy=policy, \n",
        "                       primary_metric_name='validation_acc', \n",
        "                       primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, \n",
        "                       max_total_runs=15,\n",
        "                       max_concurrent_runs=4)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Finally, let's launch the hyperparameter tuning job."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "htr = exp.submit(config=htc)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can use a run history widget to show the progress. Be patient as this might take a while to complete."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "RunDetails(htr).show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "htr.wait_for_completion(show_output=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "assert(htr.get_status() == \"Completed\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Find and register best model <a class=\"anchor\" id=\"register-model\"></a>\n",
        "When all the jobs finish, we can find out the one that has the highest accuracy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "best_run = htr.get_best_run_by_primary_metric()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now let's list the model files uploaded during the run."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "print(best_run.get_file_names())"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can then register the folder (and all files in it) as a model named `tf-dnn-mnist` under the workspace for deployment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "model = best_run.register_model(model_name='tf-dnn-mnist', model_path='outputs/model')"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Warm start a Hyperparameter Tuning experiment\n",
        "Often times, finding the best hyperparameter values for your model can be an iterative process, needing multiple tuning runs that learn from previous hyperparameter tuning runs. Reusing knowledge from these previous runs will accelerate the hyperparameter tuning process, thereby reducing the cost of tuning the model and will potentially improve the primary metric of the resulting model. When warm starting a hyperparameter tuning experiment with Bayesian sampling, trials from the previous run will be used as prior knowledge to intelligently pick new samples, so as to improve the primary metric. Additionally, when using Random or Grid sampling, any early termination decisions will leverage metrics from the previous runs to determine poorly performing training runs. \n",
        "\n",
        "Azure Machine Learning allows you to warm start your hyperparameter tuning run by leveraging knowledge from up to 5 previously completed hyperparameter tuning parent runs. In this example, we shall warm start from the initial hyperparameter tuning run in this notebook"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "warm_start_parents_to_resume_from=[htr]\n",
        "\n",
        "warm_start_htc = HyperDriveConfig(run_config=src, \n",
        "                       hyperparameter_sampling=ps, \n",
        "                       policy=policy, \n",
        "                       resume_from=warm_start_parents_to_resume_from, \n",
        "                       primary_metric_name='validation_acc', \n",
        "                       primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, \n",
        "                       max_total_runs=20,\n",
        "                       max_concurrent_runs=4)\n",
        "\n",
        "warm_start_htr = exp.submit(config=warm_start_htc)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can use the run history widget to show the progress of this warm start run. Be patient as this might take a while to complete."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "RunDetails(warm_start_htr).show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "warm_start_htr.wait_for_completion(show_output=True)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Find and register best model from the warm start run\n",
        "When all the jobs finish, we can find out the one that has the highest accuracy and register the folder (and all files in it) as a model named tf-dnn-mnist-warm-start under the workspace for deployment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "best_warm_start_run = warm_start_htr.get_best_run_by_primary_metric()\n",
        "warm_start_model = best_warm_start_run.register_model(model_name='tf-dnn-mnist-warm-start', model_path='outputs/model')"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Resuming individual training runs in a hyperparameter tuning experiment\n",
        "\n",
        "In the previous section, we saw how you can warm start a hyperparameter tuning run, to learn from a previously completed run. Additionally, there might be occasions when individual training runs of a hyperparameter tuning experiment are cancelled due to budget constraints or fail due to other reasons. It is now possible to resume such individual training runs from the last checkpoint (assuming your training script handles checkpoints). Resuming an individual training run will use the same hyperparameter configuration and mount the storage used for that run. The training script should accept the \"--resume-from\" argument, which contains the checkpoint or model files from which to resume the training run. \n",
        "\n",
        "You can also resume individual runs as part of an experiment that spends additional budget on hyperparameter tuning. Any additional budget, after resuming the specified training runs is used for exploring additional configurations.\n",
        "\n",
        "In this example, we will resume one of the child runs cancelled in the previous hyperparameter tuning run in this notebook"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "cancelled_child_runs = []\n",
        "for child_run in htr.get_children():\n",
        "    if child_run.status == \"Canceled\":\n",
        "        cancelled_child_runs.append(child_run)\n",
        "        \n",
        "if len(cancelled_child_runs) != 0:\n",
        "    child_runs_to_resume=[cancelled_child_runs[0]]\n",
        "else:\n",
        "    child_runs_to_resume=[]"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Next, we will configure the hyperparameter tuning experiment to warm start from the previous experiment and resume individual training runs and submit this warm start hyperparameter tuning run."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "resume_child_runs_htc = HyperDriveConfig(run_config=src, \n",
        "                       hyperparameter_sampling=ps, \n",
        "                       policy=policy, \n",
        "                       resume_child_runs=child_runs_to_resume, \n",
        "                       primary_metric_name='validation_acc', \n",
        "                       primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, \n",
        "                       max_total_runs=10,\n",
        "                       max_concurrent_runs=4)\n",
        "\n",
        "resume_child_runs_htr = exp.submit(config=resume_child_runs_htc)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        " We can use the run history widget to show the progress of this resumed run. Be patient as this might take a while to complete."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "RunDetails(resume_child_runs_htr).show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "resume_child_runs_htr.wait_for_completion(show_output=True)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Find and register best model from the resumed run\n",
        "When all the jobs finish, we can find out the one that has the highest accuracy and register the folder (and all files in it) as a model named tf-dnn-mnist-resumed under the workspace for deployment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "best_resume_child_run = resume_child_runs_htr.get_best_run_by_primary_metric()\n",
        "resume_child_run_model = best_resume_child_run.register_model(model_name='tf-dnn-mnist-resumed', model_path='outputs/model')"
      ]
    }
  ],
  "metadata": {
    "authors": [
      {
        "name": "nagaur"
      }
    ],
    "category": "training",
    "compute": [
      "AML Compute"
    ],
    "datasets": [
      "MNIST"
    ],
    "deployment": [
      "Azure Container Instance"
    ],
    "exclude_from_index": false,
    "framework": [
      "TensorFlow"
    ],
    "friendly_name": "Hyperparameter tuning and warm start using the TensorFlow estimator",
    "index_order": 1,
    "kernelspec": {
      "display_name": "Python 3.8 - AzureML",
      "language": "python",
      "name": "python38-azureml"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    },
    "tags": [
      "None"
    ],
    "task": "Train a deep neural network"
  },
  "nbformat": 4,
  "nbformat_minor": 2
}